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METHOD AND APPARATUS TO EXTEND THE EFFECTIVE DYNAMIC 
RANGE OF AN IMAGE SENSING DEVICE 

FIELD OF THE INVENTION 

The invention relates generally to the field of image capture, and 
more specifically to a method of extending the effective dynamic range of an 
image sensing device. 

BACKGROUND OF THE INVENTION 

Image sensing devices, such as a charge-coupled device (CCD), are 
commonly found in such products as digital cameras, scanners, and video 
cameras. These image sensing devices have a very limited dynamic range when 
compared to traditional negative film products. A typical image sensing device has 
a dynamic range of about 5 stops. This means that the exposure for a typical scene 
must be determined with a fair amount of accuracy in order to avoid clipping the 
signal. In addition, oftentimes the scene has a very wide dynamic range as a result 
of multiple illuminants (e.g. frontlit and backlit portions of a scene). In the case of 
a wide dynamic range scene, choosing an appropriate exposure for the subject 
often necessitates clipping data in another part of the image. Thus, the inferior 
dynamic range of an image sensing device relative to silver halide media results in 
lower image quality for images obtained by an image sensing device. 

An increase in the dynamic range of an image sensing device 
would allow images from digital cameras to be rebalanced to achieve a more 
pleasing rendition of the image. Also, increasing the dynamic range of an image 
sensing device would allow for more pleasing contrast improvements to the 
image, such as is described by Lee et al. in commonly assigned U.S. Patent No. 
5,012,333. 

U.S. Patent No. 6,040,858 (Ikeda) provides a complete description 
of the problem of the limited dynamic range of image sensing devices. In addition, 
Ikeda describes methods of extending the dynamic range of an image sensing 



device by utilizing multiple image signals, each with different responses to 
exposure. These multiple signals are combined by using thresholds which 
determine which signal is of higher quality at each position in the image signal to 
form an image signal having extended dynamic range. Ikeda improves upon these 
5 methods by describing a method by which these thresholds are determined for 
each color. 

However, these prior art methods, including Ikeda, require multiple 
image signals to form an image signal having extended dynamic range. Attaining 
such multiple signals can cause difficulty. For example, if the multiple image 

1 0 signals are not captured simultaneously, objects moving in the scene or motion of 
the camera may produce artifacts in an image signal having extended dynamic 
range. Additionally, if the image signals are captured simultaneously but on 
separate image capture devices, then a correspondence problem exists. Moreover, 
the additional hardware adds undesirable cost to the imaging system. 

1 5 Thus there exists a need to improve upon the method of the prior 

art in order to improve the dynamic range of an image sensing device. 
Specifically, there exists a need to produce an extended dynamic range image 
signal with a single image sensing device and a single image signal. 

20 SUMMARY OF THE INVENTION 

The present invention is directed to overcoming one or more of the 
problems set forth above. Briefly summarized, according to one aspect of the 
present invention, an image capture system generates an extended effective 
dynamic range from a signal provided by an image sensor by utilizing an image 

25 sensing device having standard photosites with a predetermined response to a light 
exposure and non-standard photosites with a slower response to the same light 
exposure. An optical section exposes the image sensing device to image light, 
thereby causing the image sensing device to generate an image signal and a 
processing section expands the response of the standard photosites to increased 

30 light exposures by utilizing the image signals from neighboring non-standard 



-3- 

photosites. Furthermore, the processing section may expand the response of the 
non-standard photosites to decreased light exposures by utilizing the image signals 
from neighboring standard photosites. 

From another aspect, the invention includes an image sensor for 
5 generating an image signal useful in an extended dynamic range system, wherein 
the image sensor includes an array of photosites divided into standard photosites 
and non-standard photosites and a structural element overlying the photosites and 
providing the standard photosites with a predetermined standard response to a 
light exposure and the non-standard photosites with a slower response to the same 
1 0 light exposure. This structural element may take the form of (a) an array of 
lenslets overlying the photosites, wherein the lenslets overlying the standard 
photosites are structured to be more efficient in focusing light than the lenslets 
overlying the non-standard photosites, (b) a mask with apertures overlying the 
photosites, wherein the apertures overlying the standard photosites are larger than 
1 5 the apertures overlying the non-standard photosites, or (c) a neutral density filter 
overlying the photosites, wherein the portion of the neutral density filter overlying 
the standard photosites is more transparent than the portion of the neutral density 
filter overlying the non-standard photosites. 

In the present invention, selected photosites of the image capture 
20 device are designed to have non-standard response to exposure. These non- 
standard photosites generally have a slower response to exposure than do the non- 
selected, or standard, photosites. The advantage of the invention is that the image 
signal from such a image capture device is processed to take advantage of the 
dynamic ranges of all photosites. Thus, an image signal having increased dynamic 
25 range is produced by interpolating the values of neighboring photosites for those 
photosites that are saturated or at a noise level. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram of an extended-range image sensing 
device according to the invention. 



Fig. 2 is a graph illustrating the response of a standard photosite 
and a non-standard photosite. 

Fig. 3A illustrates the arrangement of non-standard photosites and 
standard photosites on a panchromatic image sensing device. 
5 Fig. 3B illustrates the arrangement of non-standard photosites and 

standard photosites on a color image sensing device. 

Fig. 4. Shows an exploded block diagram of the dynamic range 
extending filter array (DREFA) processor shown in Fig. 1. 

Fig. 5 is a cross-section of an interline image sensor employing an 
1 0 array of lenslets to alter the response of selected photosites. 

Fig. 6 is a cross-section of a full frame image sensor employing a 
metal mask to alter the response of selected photosites. 

Fig. 7 is a cross-section of an image sensor employing an array of 
neutral density filters to alter the response of selected photosites. 

15 

DETAILED DESCRIPTION OF THE INVENTION 

Because imaging devices employing electronic sensors are well 
known, the present description will be directed in particular to elements forming 
part of, or cooperating more directly with, apparatus in accordance with the 
20 present invention. Elements not specifically shown or described herein may be 

selected from those known in the art. Note that as used herein, the term image is a 
two dimensional array of values. An image may be a two dimensional subset of 
another image. 

Referring to Fig. 1, light from an object or scene is incident upon a 
25 lens 2. An optical low pass filter 6 performs a slight blurring of the image in order 
to reduce the occurrence of aliasing. The image falls on an image sensing device 
10 such as a charged-coupled device (CCD). Note that other devices, such as 
CMOS devices, may be used as the image sensing device 10. 

An A/D converter 14 converts the image signal from the image 
30 sensing device 10 into a digital signal. More specifically, the A/D converter 14 
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converts the linear voltage signal from the image sensing device 10 to a discrete 
digital signal, preferably a 10 bit signal. Thus, the linear encoded values range 
from 0 to 1023. The A/D converter 14 also preferably performs processing to 
convert the linear 10 bit signal to an 8 bit logarithmic signal, as is commonly 
5 performed in the art. The following equation is used to convert the 10 bit linear 
signal a(x,y), where (x,y) specifies the row and column index of the signal location 
with reference to the image sensing device 10, into the 8 bit logarithmic signal 
b(x,y): 

J 0 0< a(x,y) <31 

b(x,y) = | 73 5975 ln 3,) _ 255 32 < a(x,y) < 1024 

1 0 Note that each stop of exposure (in the linear response region of the image sensing 
device) results in a doubling of the linear signal a(x,y) and results in an increase of 
the logarithmically encoded signal b(x,y) by 51. In this case, the value 51 
represents the number of code values per stop (cvs) of exposure. 

The system controller 18 determines on the basis of user input or 
15 on the basis of the image signal output from the A/D converter 14 whether the 
dynamic range needs to be expanded (i.e. with the "expanding mode" of 
processing) or whether there is no need for expanding ( i.e., the "normal mode" of 
processing). The system controller 18 then diverts the digital image signal b(x,y) 
to either the dynamic range extending filter array (DREFA) processor 22 if the 
20 system controller 18 is in an expanding mode, or to the color filter array (CFA) 
interpolator 26 if the system controller 18 is in a normal mode. Alternatively, the 
system controller 18 may be set at the time of manufacture to always be in the 
"expanding mode." 

In the "normal mode", the system controller 18 diverts the image 
25 signal output from the A/D converter 14 to the CFA interpolator 26. The purpose 
of the CFA interpolator 26 is to generate a full description of the color for each 
location of the digital image. In the preferred embodiment, the image sensing 
device 10 consists of an array of photosensitive elements. Each photosite is 
typically coated with either a red, green, or blue filter, as described by Bayer in 
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commonly assigned U.S. Patent No. 3,971,065, which is incorporated herein by 
reference. The Bayer array is a color filter array in which green filters are located 
in a checkerboard pattern over the photosites with red and blue filter alternating 
line by line to fill the interstices of the checkerboard pattern; this produces twice 

5 as many green filter sites as either red or bine filter sites. Note that the method 
described herein may be easily extended to color filter arrays with different 
arrangements of the primaries, a different number of primaries, or a different set of 
primaries. Thus, in the preferred embodiment, each photosite is sensitive to either 
red, green, or blue light. However, it is desirable to obtain a value of exposure for 

10 each of the red, green, and blue exposures at each photosite location. In this 

description, "red", "green", and "blue" represent the primaries of an image sensing 
device 10, as is well known in the art of image processing. A CFA interpolator 26 
generates from the image signal output from the A/D converter 14 an interpolated 
image signal consisting of a value for each of the primaries of a photosensitive 

1 5 element. For example, if a particular photosite is coated with a red filter, then the 
A/D converter 14 outputs a red level of exposure for that photosite since the red 
filter essentially blocks green and blue light from reaching the image sensing 
device 10. 

The operation of the CFA interpolator 26 is to determine the levels 
20 of exposure for a red photosite for both the green and the blue primaries. 

Similarly, the CFA interpolator 26 determines the green and red exposure levels 
for the blue photosites, as well as the red and the blue exposure levels for the 
green photosites. Generally, the CFA interpolator 26 operates by considering the 
exposure values of the photosite and the values of surrounding photosites. While 
25 any commonly known interpolator may be used, a description of a preferred CFA 
interpolator is contained in commonly assigned U.S. Patent No. 5,652,621, entitle 
"Adaptive color plane interpolation in single sensor color electronic camera", 
which is incorporated herein by reference. This patent describes apparatus for 
processing a digitized image signal obtained from an image sensor having color 
30 photosites aligned in rows and columns that generate at least three separate color 
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values but only one color value for each photosite location, and a structure for 
interpolating color values for each photosite location so that it has three different 
color values. The apparatus generates an appropriate color value missing from a 
photosite location by the interpolation of an additional color value for such 
5 photosite locations from color values of different colors than the missing color 
value at nearby photosite locations. The apparatus also obtains Laplacian second- 
order values, gradient values and color difference bias values in at least two image 
directions from nearby photosites of the same column and row and selects a 
preferred orientation for the interpolation of the missing color value based upon a 
10 classifier developed from these values. Finally, the missing color value from 
nearby multiple color values is selected to agree with the preferred orientation. 

In the "expanding mode" the system controller 18 diverts the image 
signal output from the AID converter 14 to the DREFA processor 22 in order to 
expand the dynamic range of the image signal. In the preferred embodiment, the 
1 5 dynamic range of the image sensing device 10 is expanded by selecting certain 
photosites of the image sensing device 10 to have a non-standard response. The 
arrangement of the selected photosites with respect to the image sensing device 10 
will be discussed in greater detail hereinbelow. In the preferred embodiment, the 
responses of selected photosites are slowed by altering the gain of the selected 
20 photosites, herein referred to as non-standard photosites. Altering the gain of a 
photosite is commonly practiced in the art of digital camera design and 
manufacture. 

With reference to Fig 5, it is a common practice in the art of image 
sensor manufacture to place resin lenslets 51 on top of each photosite. For 

25 example, particularly when the image sensing device 10 is an interline solid state 
image sensing device, one lenslet technique is described in US Patent No. 
4,667,092, entitled "Solid-state image device with resin lens and resin contact 
layer", which is incorporated herein by reference. In this patent, more specifically, 
a solid-state image device includes an image storage block having a block surface 

30 and a plurality of storage elements are embedded along the block surface to store 
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an image in the form of electric charge. An overlying layer is deposited to form an 
array of optical lenses in correspondence to the storage elements. An intermediate 
layer is laid between the block surface and the overlying layer. Incident light 
focuses through the lenses and the intermediate layer onto the storage elements. 
5 The intermediate layer serves as an adjusting layer for adjusting a focal length. 

Fig 5 shows a cross section of an interline solid state image sensing 
device. Without the lenslets 51, the signal readout area associated with each 
photosensitive area 55 of a photosite makes it impossible to use the whole area of 
the semiconductor substrate as the photoelectric transducer area. The conventional 
1 0 solid-state image device does not effectively utilize all incident rays thereon and 
therefore has low sensitivity. The addition of a resin lenslet 51 on top of a 
photosite allows the incident rays of light to be focused on the photoactive areas of 
the photosite, thereby more effectively utilizing the incident rays of light and 
increasing the sensitivity of the photosite. Thus, by varying the size and/or 
1 5 efficiency of the lenslet 5 1 , the sensitivity (gain) of the photosite may be easily 

altered. Thus, for interline devices and for CMOS sensors the preferred method of 
altering the gain of the photosite is by altering the lenslet 5 1 placed on top of the 
photosite. As shown in Figure 5, the location 52 has no lenslet, and therefore 
fewer incident rays of light are incident with the photosensitive area. 
20 Alternatively, a lenslet could be manufactured at location 52 with a different 

radius, shape, size or material as compared with the lenslet 51, thereby structured 
to be less efficient at focusing incident rays of light onto the photosensitive area 
55 than is the standard lenslet 51. Those skilled in the art will recognize that if the 
lenslet 51 focuses 80% of the incident rays of light onto a photosensitive area 55 
25 and the region 52 having no lenslets (or alternatively non-standard lenslets) allows 
20% of the incident rays of light onto a photosensitive area 55, then the photosite 
covered by lenslet 5 1 is 2 stops faster than the region 52. In this case, the lenslet 
51 is used for standard photosites and no lenslet is used for non-standard 
photosites, as represented by region 52. 
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With reference to Fig. 6 showing a cross section of a full frame 
image sensing devicelO, in the case where the image sensing device 10 is a Ml 
frame device, light rays incident to the photosensitive area 55 of a photosite must 
pass through an aperture of a mask, typically made from metal, which is shown in 
5 cross-section in Figure 6 to comprise light-blocking metallic mask portions 54 and 
open apertures 56 and 57 interspersed among the metallic portions. In the 
preferred embodiment, the gain of photosites may be altered by modifying the 
metal mask 54 light shield. The sensitivity of the photosite is then directly related 
to the aperture of the metal mask 54 light shield. For example, one photosite with 
10 an aperture 50% of the size of a second photosites aperture will have a response of 
50% compared to that on the second photosite. For example, a first aperture 56 of 
a light shield 54 allows 80% of the light rays incident upon that photosite to pass 
through, but a second aperture 57 is smaller and allows only 20% of the incident 
light rays to pass. Those skilled in the art will recognize that the photosite with the 
15 larger first aperture 56 is 2 stops faster than a photosite with the smaller second 
aperture 57. In this case, the first aperture 56 is used for standard photosites, and 
the second aperture 57 is used for the non-standard photosites. Thus, the aperture 
of a light mask may be modified to adjust the response of the selected photosites. 
Kodak makes full frame image sensing devices with a metal mask light shield that 
20 reduces the pixel active area of all pixels from about 80% to about 20% (for 

dithered scanner applications where the sensor is moved by 1/2 the pixel spacing 
horizontally and vertical and 4 pictures are taken). The invention thus involves 
utilizing such mask technology, but with different sized apertures, to provide an 
image sensor with a differential response to image light. 
25 In the preferred embodiment, the response of the selected non- 

standard photosites is X% (where X<=100) that of standard photosites for the same 
exposure, as shown graphically in Fig. 2. The selected photosites have a response 
that is slowed by two stops (-log XI 100) relative to the standard photosites. In the 
preferred embodiment, X= 25. In the preferred embodiment, when the system 
30 controller 1 8 is in "normal mode", all photosites have common gain equivalent to 
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that of the standard response. Thus, the image sensing device 10 consists of 
multiple sets of photosites, the standard photosites and the non-standard 
photosites. The collection of the outputs of the standard photosites constitutes a 
sparsely sampled version of a scene. Likewise, the collection of the outputs of the 
5 non-standard photosites constitutes another sparsely sampled version of a scene. 



photosites can be slowed by the use of a neutral filter coating the photosite. Fig. 7 
shows a cross section of an image sensing device with a color filter array 53. Note 
that the color filter array 53a is red, 53b is green, 53c is red, and 53d is green. A 

10 layer of neutral filters 58 is contained above the color filter array 53, although the 
position of the layer of neutral filters 58 and the color filter array 53 does not 
matter. Note that the layer of neutral filters 58 only contains a neutral filter at the 
positions of selected photosites, as indicated by the neutral filter 59. In this case, 
the layer of neutral filters 58 is transparent or nearly transparent for standard 

15 photosites and contains a neutral filter 59 for non-standard photosites. For 

example, if the neutral filter 59 consists of a material that allows X% transmission 
of light, then the response of that non-standard photosite will be slowed by 



- log J stops relative to the response of the standard photosite. If a neutral 



filter 59 is used to create non-standard photosites, then the system controller 18 is 
20 preferably set at the time of manufacture to be in "expanding mode." 



signal with an increased dynamic range by processing the digital image signal 
while considering the standard and non-standard photosites. The operation of the 
DREFA processor 22 will be described in detail hereinbelow. Accordingly, the 
25 output of the DREFA processor 22 is an expanded image signal having increased 
dynamic range. This expanded image signal is then input to the CFA interpolator 
for processing as previously described. 



DREFA processor 22 are directly connected, this is not a requirement for the 
30 present invention. The DREFA processor 22 may reside in hardware or software 



As another alternative, the responses of the selected non-standard 




The purpose of the DREFA processor 22 is to create a digital image 



Note that although Fig. 1 implies that the AID converter 14 and the 
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in close proximity to the A/D converter 14 and image sensing device 10. For 
example, the DREFA processor 22 could reside directly within a digital camera. 
However, the DREFA processor 22 may also be remote from the image sensing 
device 10. For example, the image signal output from the A/D converter 14 can be 
transmitted (after compression) via a wire or wireless connection to a personal 
computing device, printer, or remote server to apply to operation of the DREFA 
processor 22. Transmission of the image signal may also include file transfer 
protocol or email. Additionally, payment via credit card or some other means may 
be required by the DREFA processor 22 from the user. 

In the preferred embodiment, 50% of the photosites of the image 
sensing device 10 are selected to have non-standard response. Those skilled in the 
art will recognize that varying the percentage of photosites which have non- 
standard response will still result in the advantages of the present invention. In the 
case of a image sensing device 10 in which all photosites have approximately 
equivalent spectral sensitivity (i.e. a pan-chromatic image sensing device), Fig. 3 A 
shows an arrangement of the non-standard photosites that will result in 
approximately 50% of all the photosites of the image sensing device 10 being of 
non-standard response. The photosites 28 with non-standard response are marked 
with an asterisk (*), while the photosites 30 having standard response are blank. 

In the case of a color image sensing device 10, Fig. 3B shows an 
arrangement wherein 50% of each type (red, green, or blue sensitive) of photosite 
has non-standard response. For example, the photosite 32 is a red photosite having 
non-standard response, the photosite 34 is a green photosite having non-standard 
response, the photosite 36 is a blue photosite having non-standard response, the 
photosite 38 is a red photosite having standard response, the photosite 40 is a 
green photosite having standard response, the photosite 42 is a blue photosite 
having standard response. 

Note that Figs. 3 A and 3B imply a regular pattern for the location 
of the non-standard photosites. While it is preferable that the non-standard 
photosites are arranged in a regular pattern, it is by no means necessary. The non- 
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standard photosites could be arranged randomly or semi-randomly over the 
surface of the image sensing device 10, and their location would be stored in some 
place accessible to the DREFA processor 22. 

The response of a standard photosite to a certain exposure and the 
5 response of a non-standard photosite to the same exposure are shown in Fig. 2. 
Note that if a level of noise n is superimposed on the response, it can easily be 
seen that the standard photosite will yield a valid signal with lower exposures 
(beginning at exposure level E) than will the non-standard photosite (which yields 

valid signal beginning at —^-E •) Alternatively, data from the non-standard 

10 photosite will be valid for higher exposure levels (up to signal level of ^~-E2 s , 

where 5 is the inherent dynamic range of a single photosite, typically 5=5 stops) 
than would the standard photosite (which produces valid response up to an 
exposure of E2 S .) Note that both the standard photosite and the non-standard 
photosite have the same range of response in stops of exposure (5), but the 

1 5 response of the non-standard photosites is preferably - log 2 (j^j sto P s slower 

than the standard photosites, as shown in Fig. 2. It is preferred that the responses 
of the standard and non-standard photosites overlap with respect to exposure. That 

is, it is preferred that - log 2 |j^j < 5. The overall dynamic range of the image 

sensing device, considering both standard and non-standard photosites, is 

20 5- log J I . In the case of the preferred embodiment, where 5=5 and X= 25, 

2 UooJ 

the overall effective dynamic range of the image sensing device 10 is 7 stops of 
exposure. 

The processing of the DREFA processor 22 may be utilized to 
extend the overall dynamic range of the image sensing device 10 by using the non- 
25 standard photosite response to reconstruct the areas in the image where very high 
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exposures occur. Likewise, the DREFA processor 22 also uses the photo sites with 
standard response to reconstruct the signal where very low exposures occur. 

Fig. 4 shows an exploded block diagram of the DREFA processor 
22. The logarithmic image signal b(x,y) output from the AID converter 14 is 
passed to the non-standard compensator 44. The purpose of the non-standard 
compensator 44 is to compensate the non-standard photosites by accounting for 
the offset in response by X stops. In the preferred embodiment, the image signal 
corresponding to the non-standard photosites are incremented by the quantity -cvs 
log (X/100), where cvs is the number of code values per stop of exposure. In the 
preferred embodiment, the quantity cvs is 51. Alternatively, if the image signal 
input to the non-standard compensator 44 is linearly related to exposure (rather 
than logarithmically), then the non-standard compensator 44 scales the image 
signal corresponding to the non-standard photosites by a factor of 100/X. Note that 
it is assumed that the locations of the non-standard photosites are known to the 
non-standard compensator 44. The output of the non-standard compensator 44 is 
an image signal i(x,y) that has been compensated at the locations of non-standard 
photosites for the difference between the non-standard photosite response in 
relation to the standard photosite response. At the locations of standard photosites, 
the value of the image signal b(x,y) output from the A/D converter 14 is identical 
to the value of the image signal i(x,y) output from the non-standard compensator 
44. Note that the image signal i(x,y) is not limited to an 8 bit range. It the preferred 
embodiment, the value of i(x,y) ranges from 0 to 357. 

Next, the image signal i(x,y) output from the non-standard 
compensator 44 is input to a non-standard thresholder 46. The purpose of the non- 
standard thresholder 46 is to determine problem locations of the non-standard 
image signal that are of low quality due to a photosite not receiving enough 
photons to produce a valid signal or due to a photosite receiving so many photons 
that the signal saturates. The image signal at these (x,y) locations is then replaced 
by calculating a new signal based upon nearby samples of the standard image 
signal in processing performed by the signal extender 50, which will be described 



-14- 



in detail hereinbelow. All (x,y) locations of the non-standard signal for which the 
corresponding values of the non-standard signal are less than a predetermined 
threshold are considered to be problem locations. In the case of the non-standard 
photosite, this predetermined threshold used for the purpose of detecting problem 
locations is referred to as the low exposure response threshold. Thus, a location 
(x,y) is considered to be a problem location if it is a non-standard photosite and if: 
i(x,y) < Tj 

where T] is predetermined. In the preferred embodiment, the value of T] 



is - cvs logJ , which in the preferred embodiment is 102. Note that the 



threshold Tj may be dependent upon the color of the photosite at location (x,y). 
Non-standard photosites that are problem locations are referred to as noise pixels, 
since the value of i(x,y) is not sufficiently about the noise level of the image 
sensing device to be useful. 



compensator 44 is input to a standard thresholder 48. The purpose of a standard 
thresholder 48 is to determine problem locations of the standard image signal that 
are of low quality. The image signal at these locations is then replaced by 
calculating a new signal based upon nearby samples of the non-standard image 
signal in processing performed by the signal extender 50, which will be described 
in detail hereinbelow. All (x,y) locations of the standard image signal for which 
the corresponding values of the standard signal are less than a predetermined 
threshold signal are considered to be problem locations. In the case of the standard 
photosite, this predetermined threshold used for the purpose of detecting problem 
locations is referred to as the high exposure response threshold. Thus, a location 
(x,y) is considered to be a problem location if it is a standard photosite and if: 
i(x,y)> T 2 

where T 2 is predetermined. In the preferred embodiment, the value of T 2 is 254. 
Note that the threshold T 2 may be dependent upon the color of the photosite at 
location (x,y). Standard photosites that are problem locations are referred to as 
saturated pixels, since the value of i(x,y) is as high as possible at these locations. 




Likewise, the image signal i(x,y) output from the non-standard 
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Th c problem locations determined by the non-standard thresholder 
46 and the problem locations determined by the standard thresholder 48 are input 
to the signal extender 50. In addition, the image signal i(x,y) output from the non- 
standard compensator 44 is also input to the signal extender 50. The purpose of 
the signal extender 50 is to replace the image signal i(x,y) values at problem 
locations (x,y) with estimates of the signal herein referred to as replacement 
values, had the inherent dynamic range of each photosite of the image sensing 
device 10 been greater. If the problem location is coincident with a non-standard 
photosite, then the replacement value is calculated from neighboring image signal 
values coincident with standard photosites. In this embodiment, the term 
"neighboring" refers to a certain spatial distance. In the preferred embodiment, the 
photosites neighboring a selected photosites are those photosites within a distance 
of 2 photosites of the selected photosite. Likewise, if the problem location is 
coincident with a standard photosite, then the replacement value is calculated from 
neighboring image signal values coincident with non-standard photosites. In the 
preferred embodiment, the color of the photosite at the problem photosite is also 
taken into account. The replacement value for any problem location is preferably 
determined only by the signal originating from neighboring photosites of the same 
color. The output of the signal extender 50 is an image signal i'(x,y) having a 
dynamic range as if captured by an image sensing device 10 having photosites 

with inherent dynamic range of S- log 2 rather than the actual innerent 

dynamic range of S for each photosite of the image sensing device 10. Note that 
for all (x,y) locations that are not problem locations, the value of i'(x,y) is 
equivalent to i(x,y). 

As an example of the processing performed by the signal extender 
50 for the Bayer CFA pattern shown in Figure 3B, if location (x,y) is a problem 
location and (x,y) is the location of a green photosite (such as photosite 34 in 
Figure 3B), then the replacement value i'(x,y) for the image signal i(x,y) is 
calculated in the following manner: 

i'(x,y) = 0.25 *D'(jc-1o^1) + i(x+l,y-l) + i(x-l,y+l) + i(x+l,y+l)] 
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Note that signal values that the calculation of i\x,y) is dependent upon are 
expected to comply with certain requirements. For example, suppose that (x,y) is a 
problem location and (x,y) is a green photosite with non-standard response. Then 
the signal levels of neighboring photosites are used to calculate replacement value 
i\x,y). However, this assumes that the signal values of each of the neighboring 
photosites are also less than T3. In the preferred embodiment, T3 = Tj. For each 
neighboring photosite that this is not the case, that signal level is omitted from the 
calculation of the replacement value i'(x,y). For example, if i(x-l, y-1) > T3, then 
the value i'(x,y) is calculated with the following formula: 

i'(x,y) = 1/3 *[i(x+l,y-l) + i(x-l,y+\) + i(x+l,y+l)] 
Generally, the interpolation scheme for determining a replacement value at 
problem location (x,y), where the location (x,y) corresponds to a green photosite 
which is also a standard photosite on a image sensing device having a Bayer 
pattern filter array is given with the following equation: 
2 ^i(x + j,y + k)W(x + j,y + k) 

fl i(x + j,y + k)>T 3 
where W(x + j,y + k) = < 

|0 otherwise 

Note that the same equation is applied to determine the replacement value if the 
problem location corresponds with a green photosite which is also a non-standard 

photosite. However, in this case W{x + j,y + k) = 

in the preferred embodiment, 7V = T_2. 

As another example, also in connection with the Bayer CFA pattern 
shown in Figure 3B, if location i(x,y) is a problem photosite and (x,y) is the 
location of a red or blue photosite, then the replacement value i'(x,y) for the image 
signal i(x,y) is calculated in the following manner: 

i'(x,y) = 0.25 *[i(x-2,y) + i(x+2,y) + i(x,y+2) + i(x,y-2)] 



l i(x + j,y + k)<T 4 

v , where 

0 otherwise 
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When location (x,y) is a red or blue photosite and is also a standard photosite, the 
equation for determining the replacement value i\x,y) may be generalized as 
follows: 



--2,0,2 £=-2,0,2 

1 i(x + j,y + k)>T 3 



5 where W(x + j,y + k) , , 

[0 otherwise 

Note that in this case, either j or k must be 0, but y and k are never both zero. 
Note also that the same equation is applied to determine the replacement value if 
the problem location corresponds with a red or blue photosite which is also a non- 
standard photosite. However, in this case 

1 0 W(x + j, y + k) = \ 1 ^ X + J ' y + ^ < Ta , where in the preferred embodiment, 
[0 otherwise 

T 4 = T 2 . 

The interpolation scheme described above for the purpose of 
generating an image signal with an extended dynamic range from more than one 
1 5 sparsely sampled image signals may be modified by those skilled in the art. 
However, many such modifications to the above interpolation scheme may be 
derived and should not be considered as significant deviations of the present 
invention. 

Those skilled in the art will recognize that the above interpolation 
20 scheme performed by the signal extender is a lowpass filter, which is well known 
in the art. Typically, the application of a lowpass filter to a digital image signal 
has a similar effect to reducing the resolution of the digital image signal. Thus, the 
processing performed by the DREFA processor 22 is a method by which the 
spatial resolution of the image sensing device 10 may be traded for dynamic range 
25 of the image sensing device 10. Indeed, those areas of an image where the 

interpolation scheme is implemented to increase the dynamic range of the signal 
appear noticeably softer (less sharp) than the image would have if that same area 
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of the image had been captured by the sensor in such a fashion that no "problem 
locations" (as defined by the non-standard thresholder 46 and the standard 
thresholder 48) occur. 

The invention has been described in detail with particular reference 

5 to certain preferred embodiments thereof, but it will be understood that variations 
and modifications can be effected within the spirit and scope of the invention. For 
example, the invention provides an image capture system that expands the 
dynamic range in both directions, i.e., that expands the response of the standard 
photosites to increased exposures by utilizing the image signals from neighboring 

1 0 non-standard photosites and expands the response of the non-standard photosites 
to decreased exposures by utilizing the image signals from neighboring standard 
photosites. It would be likewise feasible for the system to work on dynamic range 
from only one direction, i.e., to expand the response of only the standard 
photosites to increased light exposures by utilizing the image signals from 

1 5 neighboring non-standard photosites, or alternatively, to expand the response of 
only the non-standard photosites to decreased light exposures by utilizing the 
image signals from neighboring standard photosites. 
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PARTS LIST 
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lens 


6 


optical lowpass filter 


10 


image sensing device 


14 


AID converter 


18 


system controller 


22 


DREFA processor 


26 


CFA interpolator 


28 


non-standard photosite 


30 


standard photosite 


32 


red non-standard photosite 


34 


green non-standard photosite 


36 


blue non-standard photosite 


38 


red standard photosite 


40 


green standard photosite 


42 


blue standard photosite 


44 


non-standard compensator 


46 


non-standard thresholder 


48 


standard thresholder 


50 


signal extender 


51 


standard lenslet 


52 


non-standard lenslet 


53 


color filter array 


54 


metallic mask portion 


55 


photosensitive area 


56 


large aperture 


57 


small aperture 


58 


neutral density filter layer 


59 


non-standard neutral density filter 



WHAT IS CLAIMED IS: 
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1 . An image capture system for generating an extended 
effective dynamic range from a signal provided by an image sensor, said image 
capture system comprising: 

an image sensing device having standard photosites with a 
predetermined response to a light exposure and non-standard photosites with a 
slower response to the same light exposure; 

an optical section for exposing the image sensing device to image 
light, thereby causing the image sensing device to generate an image signal; and 

a processing section for expanding the response of the standard 
photosites to increased light exposures by utilizing the image signals from 
neighboring non-standard photosites. 

2. The image capture system as claimed in claim 1 wherein the 
processing section expands the response of the non-standard photosites to 
decreased light exposures by utilizing the image signals from neighboring 
standard photosites. 

3 . The image capture system as claimed in claim 2 wherein the 
processing section further comprises: 

means for processing the image signals against a plurality of 
thresholds, including a high exposure response threshold for the standard 
photosites and a low exposure response threshold for the non-standard photosites; 

means for replacing the image signals from standard photosites 
exceeding the high exposure response threshold with a combination of the image 
signals from a neighborhood of non-standard photosites; and 

means for replacing the image signals from non-standard photosites 
less than the low exposure response threshold with a combination of the image 
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signals from a neighborhood of standard photo sites, thereby producing an output 
image signal with an extended effective dynamic range. 



4. The image capture system as claimed in claim 1 wherein the 
image sensing device and the optical section are part of a digital camera and the 
processing section is part of a host computer separate from the digital camera. 

5 . The image capture system as claimed in claim 4 wherein the 
processor is accessible via a network 

6. The image capture system as claimed in claim 1 wherein the 
image sensing device, the optical section and the processing section are included 
in a digital camera. 

7. The image capture system as claimed in claim 1 wherein the 
non-standard photosites have a response that is slower by at least one stop 
compared to the standard photosites. 

8. The image capture system as claimed in claim 1 wherein the 
photosites are monochromatic photosites. 

9. The image capture system as claimed in claim 1 wherein the 
photosites are color photosites and the neighboring non-standard photosites are of 
the same color as standard photosite being processed by the processing section. 

1 0. An image capture system for generating an extended 
effective dynamic range from a signal provided by an image sensor, said image 
capture system comprising: 
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an image sensing device having standard photosites with a 
predetermined response to a light exposure and non-standard photosites with a 
slower response to the same light exposure; 

an optical section for exposing the image sensing device to image 
light, thereby causing the image sensing device to generate an image signal; and 

a processing section for expanding the response of the non-standard 
photosites to decreased light exposures by utilizing the image signals from 
neighboring standard photosites. 

1 1 . The image capture system as claimed in claim 10 wherein 
the processing section expands the response of the standard photosites to increased 
light exposures by utilizing the image signals from neighboring non-standard 
photosites. 

12. An image capture system providing an extended effective 
dynamic range, said system comprising: 

an image sensing device having standard photosites with a 
predetermined standard response to a light exposure and non-standard photosites 
with a slower response to the same light exposure; 

an optical section for exposing the image sensing device to image 
light, thereby causing the image sensing device to generate an image signal; 

means for converting the image signal into digital image signals 
corresponding to the output of the standard and non-standard photosites; and 

a processor that (a) processes the digital image signals against a 
plurality of thresholds, including a high exposure response threshold for the 
standard photosites and a low exposure response threshold for the non-standard 
photosites, (b) replaces the digital image signals from standard photosites 
exceeding the high exposure response threshold with a combination of the digital 
image signals from a neighborhood of non-standard photosites and (c) replaces the 
digital image signals from non-standard photosites less than the low exposure 
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response threshold with a combination of the digital image signals from a 
neighborhood of standard photosites, thereby producing an output digital image 
signal with an extended effective dynamic range. 

1 3 . The image capture system as claimed in claim 1 2 wherein 
the non-standard photosites have a response that is slower by at least one stop 
compared to the standard photosites. 

14. The image capture system as claimed in claim 12 wherein 
the photosites are monochromatic photosites. 

1 5 . The image capture system as claimed in claim 1 2 wherein 
the photosites are color photosites. 

16. The image capture system as claimed in claim 15 wherein 
the color photosites are arranged in a color filter array pattern. 

17. The image capture system as claimed in claim 16 wherein 
each photosite is sensitive to one of a plurality of colors and the processor 
interpolates the other colors for each photosite from the neighboring photosites. 

18. The image capture system as claimed in claim 12 wherein 
the image sensing device, the optical section and the converting means are part of 
a digital camera and the processor is part of a host computer separate from the 
digital camera. 



1 9. The image capture system as claimed in claim 1 8 wherein 
the processor is accessible via a network. 
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20. The image capture system as claimed in claim 12 wherein 
the image sensing device, the optical section, the converting means and the 
processor are included in a digital camera. 

21 . An image capture device providing an extended effective 
dynamic range, said image capture device comprising: 

an image sensing device having standard photosites with a 
predetermined standard response to a light exposure and non-standard photosites 
with a slower response to the same light exposure; 

an optical section for exposing the image sensing device to image 
light, thereby causing the image sensing device to generate an image signal; 

means for converting the image signal into digital image signals 
corresponding to the output of the standard and non-standard photosites; and 

a processor that expands the response of the standard photosites to 
increased exposures by utilizing the digital image signals from neighboring non- 
standard photosites and expands the response of the non-standard photosites to 
decreased exposures by utilizing the digital image signals from neighboring 
standard photosites. 

22. The device as claimed in claim 21 wherein the processor (a) 
processes the digital image signals against a plurality of thresholds, including a 
high exposure response threshold for the standard photosites and a low exposure 
response threshold for the non-standard photosites, (b) replaces the digital image 
signals from standard photosites exceeding the high exposure response threshold 
with a combination of the digital image signals from a neighborhood of non- 
standard photosites and (c) replaces the digital image signals from non-standard 
photosites less than the low exposure response threshold with a combination of the 
digital image signals from a neighborhood of standard photosites, thereby 
producing an output digital image signal with an extended effective dynamic 
range. 
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23. A method for generating an extended effective dynamic 
range from a signal provided by an image sensor, said method comprising the 
steps of: 

generating image signals from an image sensing device having 
standard photosites with a predetermined standard response to a light exposure 
and non-standard photosites with a slower response to the same light exposure; 
and 

expanding the dynamic range of selected photosites to extreme 
exposures by utilizing the image signals from neighboring photosites having a 
response, either standard or non-standard, opposite to that of the selected 
photosites. 

24. The method as claimed in claim 23 wherein the step of 
expanding the dynamic range comprises expanding the response of the standard 
photosites to increased exposures by utilizing the image signals from neighboring 
non-standard photosites and expanding the response of the non-standard 
photosites to decreased exposures by utilizing the image signals from neighboring 
standard photosites. 

25. The method as claimed in claim 24 wherein the step of 
expanding the respective response of the standard and non-standard photosites 
comprises the steps of: 

processing the image signals against a plurality of thresholds, 
including a high exposure response threshold for the standard photosites and a low 
exposure response threshold for the non-standard photosites; 

replacing the image signals from standard photosites exceeding the 
high exposure response threshold with a combination of the image signals from a 
neighborhood of non-standard photosites; and 
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replacing the image signals from non-standard photosites less than 
the low exposure response threshold with a combination of the image signals from 
a neighborhood of standard photosites, thereby producing an output image signal 
with an extended effective dynamic range. 

26. An image sensor for generating an image signal with a 
differential response to image light, said image sensor comprising: 

an array of photosites divided into standard photosites and non- 
standard photosites; and 

a structural element overlying the photosites and providing the 
standard photosites with a predetermined standard response to a light exposure 
and the non-standard photosites with a slower response to the same light exposure. 

27. The image sensor as claimed in claim 26 wherein the 
structural element comprises an array of lenslets overlying the photosites, wherein 
the lenslets overlying the standard photosites are structured to be more efficient in 
focusing light than the lenslets overlying the non-standard photosites. 

28. The image sensor as claimed in claim 26 wherein the 
structural element comprises a mask with apertures overlying the photosites, 
wherein the apertures overlying the standard photosites are larger than the 
apertures overlying the non-standard photosites. 

29. The image sensor as claimed in claim 26 wherein the 
structural element comprises a neutral density filter overlying the photosites, 
wherein the portion of the neutral density filter overlying the standard photosites 
is more transparent than the portion of the neutral density filter overlying the non- 
standard photosites. 



-27- 

30. An algorithm utilizing the image sensor as claimed in claim 
26 in order to expand the response of the standard photosites to increased 
exposures by utilizing the image signals from neighboring non-standard photosites 
and to expand the response of the non-standard photosites to decreased exposures 
by utilizing the image signals from neighboring standard photosites. 

3 1 . The image sensor as claimed in claim 26 further including a 
color filter array comprised of two or more colors overlying the photosites, said 
standard and non-standard photosites being associated with each color of the 
array. 

32. The image sensor as claimed in claim 3 1 wherein the color 
filter array is a Bayer array. 

33. An algorithm utilizing the image sensor as claimed in claim 
31 in order to expand the response of the standard photosites of each color to 
increased exposures by utilizing the image signals from neighboring non-standard 
photosites of the same color at least two lines removed from the corresponding 
standard photosite and to expand the response of the non-standard photosites of 
each color to decreased exposures by utilizing the image signals from neighboring 
standard photosites of the same color at least two lines removed from the 
corresponding non-standard photosite. 

34. A digital camera including the image sensor claimed in 

claim 26. 

35. The image sensor as claimed in claim 26 wherein the 
structural element comprises an array of lenslets overlying the standard photosites, 
and the non-standard photosites are not overlayed with lenslets. 
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ABSTRACT 

An image capture system generates an extended effective dynamic 
range from a signal provided by an image sensor by utilizing an image sensing 
device having standard photosites with a predetermined response to a light 

5 exposure and non-standard photosites with a slower response to the same light 
exposure. An optical section exposes the image sensing device to image light, 
thereby causing the image sensing device to generate an image signal and a 
processing section expands the response of the standard photosites to increased 
light exposures by utilizing the image signals from neighboring non-standard 

10 photosites. Furthermore, the processing section may expand the response of the 
non-standard photosites to decreased light exposures by utilizing the image signals 
from neighboring standard photosites. The differential response of the image 
sensor is provided by a structural element, such as an array of lenslets, a mask or a 
neutral density filter, overlying the photosites and providing the standard 

1 5 photosites with a predetermined standard response to a light exposure and the non- 
standard photosites with a slower response to the same light exposure. 
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